Metadaten-Verwaltung in einem verteilten RAM-basierten Speicherdienst

Traditional disk-based storage solutions face problems with fast data access to billions of small data objects, as needed by large-scale interactive web applications and online graph processing. Because of the often irregular access patterns they must keep almost all data in RAM caches, which need to be manually synchronized with secondary storage and need a lot of time to be reloaded in case of power outages. DXRAM addresses this challenge by keeping all data always in RAM of potentially many nodes aggregated in a data center. The main aims of DXRAM are support of billions of small data objects (16-64 byte) and providing persistence by a novel SSD-aware logging approach allowing to recover failed nodes very fast. Although data and meta-data management are widely researched the sheer amount of very small objects rises new problems. The use of supercomputers can lessen the problems, but is very cost-intensiv. Therefore the use of traditional PCs is preferred. This thesis presents a novel integrated approach of local and global meta-data management allowing a fast data access and high throughput while being very space-efficient. The local meta-data management includes an efficient paging-like translation scheme for global IDs to virtual memory addresses and a memory management optimized for many small data objects. A novel memory allocator allows to reduce the meta-data for allocations down to two bytes per object, allowing to store over one billion objects per node (with 32 GB of memory per node). Changing data granularities is supported by an efficient incremental defragmentation which can run in the background or as needed parallel to the overall system. A super-peer-overlay is used for a range-based meta-data management allowing fast node lookups while being space-efficient by combining object IDs in ranges. The super-peers manage these ranges together with backup-node information to support parallel and fast recovery of meta data and data of failed peers. Furthermore, the same concept can also be used for caching. The choosen peer-to-peer approach allows to use the storage system as self-contained backend storage or to run application code on the storage nodes. Both local and global meta-data management have been successfully evaluated and compared with state-of-the-art approaches and systems and the results show the high throughput and the very efficient memory usage, which are better than traditional systems.